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(54) Animation creation apparatus and method 



(57) A sound data extraction section (1) extracts a 
predetermined parameter from sound data. A figure 
memory (5) stores a plurality of character models. A mo- 
tion data memory (6) stores motion data of each of the 
plurality of character models. A location data memory 
(3) stores kx;ation data of each of the plurality of char- 
acter models in virtual space. An indication section (11) 



indicates a change in the location data of at least one of 
the plurality of character models by a user. An editing 
section (4) changes the motion data of each of the plu- 
rality of character models to be in cooperation with each 
other according to the predetermined parameter and the 
changed location data. A display section (10) displays 
the plurality of character models each moving according 
to the changed motion data. 
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Description 

FIELD OF THE INVENTION 

[0001] The present invention relates to an animation 
creation apparatus and a method to control a video im- 
age of a character model according to sound data. 

BACKGROUND OF THE INVENTION 

[0002] Because of the spread and high ability of per- 
sonal computers, a personal user can easily enjoy and 
operate computer graphics and computer music. In this 
situation, each user desires an interactive tool to simul- 
taneously enjoy audio and video and to easily create an 
original animation matched with the audio. 
[0003] As an apparatus to create a computer art by 
sound and video, Japanese Patent Disclosure (Kokal) 
PH6-110452 is known. In this reference, an apparatus 
to detemiine graphic information according to perform- 
ance data of musical instruments is disclosed. First, mu- 
sical performance data of a digital sound source such 
as MIDI standard is Inputted. MIDI standard represents 
a sheet music by binary code. From the musical per- 
formance data, a position, a color, a shape, a size, a 
direction, and a change speed of figure to be displayed 
is automatically determined. As a result, the computer 
art Including the figure Is created according to these da- 
ta. 

[0004] As an apparatus to create a computer anima- 
tion by sound and video, Japanese Patent Disclosure 
(Kokai) PH9-91465 is known. In this reference, the fig- 
ure and the motion data are interactively related with the 
sound by the user. In proportbn to change of the sound, 
the figure is moving by changing the motbn data related 
with the sound data. In this invention, the user can edit 
a relation between the sound, the figure and the motion 
data while playing the music. This edition is. for exam- 
ple, a change of the relation or an addition or a deletion 
of the relation. As a result, the user can create various 
kinds of animation for one music. 
[0005] In this kind of apparatus of prior art, a relation 
between the sound data, the figure data, and the motion 
data is predetermined by a calcuiatton equation or a cor- 
respondence table previously prepared. As bng as the 
musical performance method is not changed, only one 
video is created from one music. Furthermore, a plurality 
of figures to be displayed are independently moved by 
the motion previously assigned to the sound. Therefore, 
the animatnn in which the plurality of figures are in co- 
operation with each other is not created 
[0006] As mentioned-above, in the animation creation 
apparatus of the prior art, various animation is not inter- 
actively edited / created by the user's feeling. Further- 
more, the video of organized dance, i.e., a plurality of 
objects cooperatively moving to the music, is not creat- 
ed. In short, an animation creation environment includ- 
ing a low free degree of edition and a tow representation 



power is only presented to the user. 

SUMMARY OF THE INVENTION 

s [0007] It is an object of the present invention to pro- 
vide an anInfBtion creation apparatus and method to co- 
operatively move a plurality of figures in space and time 
to music. 

[0008] According to the present invention, there is 

10 provided an animation creation apparatus, comprising: 
sound data extraction means for extracting predeter- 
mined parameter from sound data; figure memory 
means for storing a plurality of character models to be 
displayed; motion data memory means for storing mo- 

is tion data of each of the plurality of character models; 
location data memory means for storing location data of 
each of the plurality of character models in virtual space 
to be displayed; indication means for indicating change 
of the location data of at least one of the plurality of char- 

20 acter models; editing means for changing the motion da- 
ta of each of the plurality of character models to .be in 
cooperation with each other according to the predeter- 
mined parameter and the changed location data; and 
display means for displaying the plurality of the charac- 

25 ter models each moving according to the changed mo- 
tion data. 

[0009] Further in accordance with the present inven- 
tion, there is also provided an animation creation appa- 
ratus, comprising: figure memory means for storing at 

30 least one character model to be displayed; motion data 
memory means for storing motion data of the one char- 
acter model; indication means for indicating motion of 
the one character; editing means for moving the one 
character according to the motion data in response to 

35 the indication; object image memory means for storing 
a plurality of object images to compound on the one 
character; image composition means for selectively 
compounding each of the plurality of object images on 
the one character in synchronization with motion of the 

40 one character; and display means for displaying the mo- 
tion of the one character on which the object image is 
compounded. 

BRIEF DESCRIPTION OF THE DRAWINGS 

45 

[0010] Fig. 1 is a block diagram of an animatton cre- 
ation apparatus according to a first embodiment of the 
present invention. 

[0011] Fig. 2 is a schematic diagram showing con- 
so structbn of a human model. 

[001 2] Fig. 3 is a schematic diagram showing a mem- 
ory fomr^t in a figure data memory section in Fig. 1. 
[001 3] Fig. 4 is a schematic diagram of an example of 
a human model as the figure data. 
5S [001 4] Fig. 5 is a block diagram of a motion data cre- 
ation section 6 in Fig. 1 . 

[001 5] Fig. 6 is a schematic diagram of an example of 
the human model to which the user indicates a pose. 
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[0016] Fig. 7 is an example of a display of a motion 
data editing section of the first embodiment. 
[0017] Fig. 8 is an example of a display of a motion 
created by the motion data editing section 82. 
[0018] Fig. 9 is an example of pose data in a pose file. 
[0019] Fig. 10 is an example of motion data in a mo- 
tion file. 

[0020] Fig. 1 1 is an example of location data on an XY 
plane. 

[0021] Fig. 12 is an example of location of character 
model on an XY plane. 

[0022] Fig. 1 3 is a block diagram of the editing section 
of the first embodiment. 

[0023] Fig, 1 4 is an example of a display of a motion 
data relation interface as local location data in the first 
embodiment. 

[0024] Fig. 1 5 is an example of a display of a nrK>tion 
data relation interface as general location data in the 
first embodiment. 

[0025] Figs. 1 6A and 16B are examples of moving lo- 
cation of character models on an XY plane. 
[0026] Fig 1 7 is a flow chart of processing of locatbn 
change of character model in the first embodiment. 
[0027] Fig. 18 is an example of three-dimensional lo- 
cations of character models. 
[0028] Fig. 1 9 is a flow chart of processing of a motion 
data change section 44 in the first embodiment. 
[0029] Fig. 20 is a first example of the location of three 
character models. 

[0030] Fig. 21 is a second example of the location of 
three character models. 

[0031] Fig. 22 is a third example of the location of 
three character models. 

[0032] Fig. 23 is a flow chart of processing of a motion 
generation section 45 in the first embodiment. 
[0033] Fig. 24 is a time chart representing a relation 
between the motion and a tempo in the first embodi- 
ment. 

[0034] Fig. 25 is a block diagram of the editing section 
4 in a first modification of the first embodiment. 
[0035] Fig. 26 is an example of a display of an inter- 
face in sound data / motion data relation section 46 in 
the first modification. 

[0036] Fig. 27 is a flow chart of processing of the mo- 
tion generation section 45 in the first modification. 
[0037] Fig. 28 is a time chart representing a relation 
between a derived motion and the tempo in the first em- 
bodiment. 

[0038] Fig. 29 is a block diagram of the editing section 
4 in a second modification in the first embodiment. 
[0039] Fig. 30 is an example of a display of an inter- 
face of an operation / motion data relatbn section 47 in 
the second modification. 

[0040] Fig. 31 is a flow chart of processing of the mo- 
tion data change section 44 in the second modification. 
[0041] Fig. 32 is an example of a display presenting 
three human models. 

[0042] Fig. 33 is a block diagram of the animatbn cre- 



ation apparatus according to a second embodiment of 
the present inventton. 

[0043] Fig. 34 is an example of face image data in a 
face Image memory section 50 in the second embodi- 
s ment. 

[0044] Fig. 35 is an example of a memory format in a 
face movie file of the face image memory section 50. 
[0045] Fig. 36 is a block diagram of the editing section 
4 In the second embodiment. 
10 [0046] Fig. 37 is an example of a display of an inter- 
face of a figure data selection section 41 in the second 
embodiment. 

[0047] Figs. 38A. 38B and 380 are examples of the 
face image put on a head plate by a figure change sec- 

is lion 51 in the second embodiment. 

[0048] Fig. 39 is a flow chart of processing of the figure 
change section 51 in the second embodiment. 
[0049] Fig. 40 is an example of the back image of the 
head put on the head plate of the human model in the 

20 second embodiment. 

[0050] Fig. 41 is an example of a cloth image put on 
a body plate of the human model in the second embod- 
iment. 

[0051] Fig. 42 is a block diagram of the animation cre- 
2S atlon apparatus according to a third embodiment of the 
present invention. 

DETAILED DESCRIPTION OF THE PREFERRED 
EMBODIMENT 

30 

[0052] Fig. 1 is a block diagram of the animation cre- 
ation apparatus according to the first embodiment of the 
present invention. The animation creation apparatus in- 
cludes a sound data extraction section 1 , a sound output 
3S section 2, a location data memory sectbn 3, an editing 
section 4, a figure data memory section 5, a motion data 
memory section 6, a pose data memory section 7, a mo- 
tion data creation section 8, a drawing section 9, a dis- 
play section 10. an indication sectbn 11. 
40 [0053] For example, a Mi Dl signal is input to a digital 
sound source based on MIDI standard. The sound data 
extraction section 1 supplies the MIDI signal to the 
sound output section 2. In addition to this, the sound 
data extraction section 1 extracts sound data such as a 
45 tune perfomnance data from the MIDI signal, and sup- 
plies the sound data to the editing section 4. The MIDI 
signal represents information to drive the digital sound 
source as serial binary code, consisting of tempo data 
for music progress, channel data for tone, musical inter- 
so val data for musical pitch, musical intensity data for mu- 
steat strength power, and effect sound data for musical 
effect. In the first embodiment, an example using the 
tempo data is explained. 

[0054] The figure data memory section 5 previously 
ss stores figure data such as a character model to be dis- 
played. The figure data includes not only two-dimen- 
sbnal figure data but also three-dimensional figure data. 
As a format of the three-dimensional figure data, in the 
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first embodiment, X file format of Direct 3D (Microsoft 
corporation) as standard of three-dimensional computer 
grapliics API Is explained. As the three-dimensional fig- 
ure data, the human character model is used. Objects 
such as head, arms, and legs ol the human model are s 
prepared as a file of X file format. In addition to this, a 
human body file ( chr) to recompound the objects as a 
human body is prepared. 

[0055] As shown In Fig. 2, the human model consists 
of fifteen objects as a hierarchical structure. In Fig. 2, a 
hip of the human model is set as a center point (root). 
In this hierarchical structure, when the hip is rotated, 
child objects belonged to the hip are dependently rotat- 
ed. As a result, the whole body of the human model ro- 
tates. 

[0056] Fig. 3 shows an example of the human body 
file (model #1. chr) and Fig. 4 shows an example of a 
display of the human model. In Fig. 3, as for fifteen ob- 
jects of the human model, X file name and the origin 
conversion data to recompound are described. At a first 
line in Fig. 3, hip data of the human model is described. 
X file name is hip. X. and the origin conversion data is 
(0.0, 5.35, 0.0). At a second line in Fig. 3, a left leg 1 (X 
file name is leg LI . X) is based on the hip as the parent 
object. By using the origin conversion data (0.5, -0.85, 
0.0), the left leg 1 is moved while connecting to the hip. 
At a third line in Fig. 3, a left leg 2 (X file name is leg L2. 
X) is based on the origin of the left leg 1 as the parent 
object. By using the origin conversion data (0.0, -2, 0.0), 
the left leg 2 is moved while connecting to the left leg 1 . 
In the first embodiment, the three-dimensional figure da- 
ta is not limited to the X file format. DXF format in CAD 
region may be used. Othenvise, data format of three- 
dimensional figure created by various modeling soft- 
ware may be converted to X file format. 
[0057] The motion data creation section 8 creates the 
motion data of the figure stored in the figure data mem> 
ory section 5. In the first embodiment, the motion of the 
figure is created by a method of key frame animation. 
Therefore, different poses of one figure are created as 
key frames. The different poses are connected by ar- 
ranging in order and presen/ed as a motion file. 
[0058] Fig. 5 is a block diagram of the motion data 
creation section 6. In Fig. 5, a pose creation section 81 
reads the figure data stored In the figure data memory 
sectbn 5, and assigns a pose to the figure. Fig. 6 is an 
example of a display of the pose assigned to the human 
model. A user can select each object of the human mod- 
el by a mouse. The user assigns the pose to the selected 
object by key input operation such as an arrow key and 
a shift key of the indication section. For example, the 
selected object is converted as rotation and movement 
for X, Y, Z axis by the key input operation. As an object 
selection by the mouse, a detection of a cross point be- 
tween a bounding box of the figure and a point of the 
mouse is executed as a picking operation. In Fig. 6, a 
left ami 1 of the human model Is selected as a square 
part and rotated about the Z axis. As mentioned-above, 
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each object of the human model is connected as hier- 
archk;al structure. Therefore, if the left arm 1 is rotated 
as shown in Fig. 6, a left arm 2 and a left hand as the 
child object are dependently rotated. The pose data cre- 
ated by this operation is stored as a pose file (extension 
is Vpse') in the pose data memory section 7. The user 
can retrieve the pose file and recreate the pose by the 
indication sectton 11. 

[0059] In Fig. 5, the motion data editing section 82 cre- 
ates a series of motion data of the figure by connecting 
a plurality of poses in order. Fig. 7 is an example of a 
display of an interface by the motbn data editing section 
82. In this interlace, the user can indicate the motion 
data of the human model as key frame animation. For 
example, a pose file name to store a series of motions 
and a number of division of interpolation between two 
poses are input by the user. In Fig. 7, nine poses com- 
prising a dance motion (wap) are arranged in order. A 
first pose Is called wap 1. pse, a second pose is called 
v^p 2. pse, a third pose is called wap 3. pse. In case of 
actual animation using these poses, an interpolation be- 
tween two poses is executed by the number of division 
"4". By pushing a bottom "OK" in the interface, the mo- 
tion of animation interpolated between two poses is dis- 
played in order for the user to confirm. 
[0060] Fig. 8 shows each pose as the wap dance ar- 
ranged in order As for each object (hip, arm, leg, •••) of 
the human model between two poses, data for "dir" 
(three dimensional vector of the object), 'pos' (position 
of the object) and ' up" (shift vector of the object for y 
axis) are linear-interpolated. The number of division of 
interpolation is changed through this interface. In case 
of a large number of divisions, a motion speed of the 
human model goes up. In case of a small number of di- 
vision, the motion speed of the human nrK>del goes 
down. This kind of motion data is applicable to any hu- 
man model based on composition in Fig. 2. In this inter- 
face, one pose file name is changed to other pose file 
name representing different motion of the human model 
by the user. Furthermore, by reading a motion file ex- 
plained afterward, the user can edit the motion. 
[0061] The pose data memory section 7 stores the 
pose data created by the motbn data creation section 
8 as a pose file (extension is ".pse"). Fig. 9 shows an 
example of pose data In the pose file (wap 1 . pse). At a 
first line, a header representing the pose file is de- 
scribed. At a second line, "Body" representing fifteen ob- 
jects (parts) comprising the human model is described. 
As for each " Part', an identifier (id) of the object, a three- 
dimensional vector (dir) of the. object, a shift vector (up) 
of the object for y axis, and a position (pos) of the object 
are described. In this case, the identifier "0" represents 
the hip, the identifier "1 " represents the left leg 1 and the 
identifier '2' represents the left leg 2. Each data of "dir" 
"pos' "up" are obtained when the pose of the object is 
detemnined by rotatbn and movement. 
[0062] The motion data memory section 6 stores the 
motk)n data of the figure created by the motbn data ere- 
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ation section 8. The motion data is presented as a mo- 
tion file (extension is "mtn"). In the motion file, a plurality 
of pose file names and the number of divisions indicated 
by the motion data are described. Fig. 10 shows an ex- 
ample of the motion data In the motion file (wap. mtn). 
At a first line, a header representing the motion file is 
described. At a second line, "Motion" represents that 
this motion consists of nine poses. As for each "Pose", 
order (id) of the pose, a number (div) of division of inter- 
polation, a pose file name (fname) are described. In this 
case, a first pose (id is "0") in a series of motbns is pre- 
sented as the pose file "wap 1. pse". An interpolation 
between the first pose "wap 1 . pse" and a second pose ' 
"wap 2. pse" is executed by the number of division "4". 
As mentioned-above, the motion of the object is gener- 
ated by a plurality of the pose data and the interpolated 
data between two pose data. 
[0063] The location data memory section 3 stores lo- 
cation data of a plurality of figures (objects) in virtual 
space. In the first embodiment, a plurality of objects are 
located on a two-dimensional lattice. Fig. 11 shows the 
two-dimensional lattice of "5x5". Actual area beatable 
the object is "3x3" area of inner side. The outside area 
is non-iocatable area (X). The outside area is used for 
' processing of neighboring points. The location status of 
the object on the two-dimensional lattice is represented 
as array (xjy). This location data is stored in memory 
space of the computer. Fig. 12 shows each value stored 
in the two-dimensional array [x][y] in case three objects 
A, B, C are arranged along a lateral line in which A is 
located at the center position. The value in the outside 
area non-locatable the object is '-1". The value in the 
inside area is "0" if the object is not located therein. The 
value of the lattice of the object A is " 1 the value of the 
lattice of the object B is '2"; and the value of the lattice 
of the object C is "S". In short, array [1][2]-2; array [2][2] 
=1 ; and array [3][2]=3. In this case, assume that the ob- 
ject A is moved to array [2][1]. The value ot original lat- 
tice of the object A is changed to "0", and the value of 
new lattice of the object A Is changed to "1". In short, 
array [2][2]=0. and array I2][1 ]=1 . 
[0064] The editing section 4 edits and generates var- 
- ious motion of the plurality of figures (objects) according 
to an indication from the indication section 11. For ex- 
ample, the editing section 4 selects a plurality of desired 
figures from the figure data memory section 5, selects 
a desired motion data from the motion data memory sec- 
tion 6, relates the motion data with the location data of 
the figure, changes the relation data or the location data, 
and relates the motion data with tempo data of the sound 
extracted by the sound data extraction data. Fig. 1 3 is 
a block diagram of the editing section 4. In Fig. 13, a 
figure data selection section 41 selects the figure data 
to be displayed from the figure data memory section 5. 
As the figure data, a plurality of human body files of hu- 
man models each being a dancer are prepared. For ex- 
ample, three human models are selected from these hu- 
man body files. In Fig. 1 3, a location data / motion data 



re lation section 42 selects the motion data tor the human 
model's dancing from the motion data memory section 
8, and relates the motion data with the location data of 
the human model. As an example of a relation between 
s the motion data and the location data, local location data 
and general location data are selectively used. 
[0065] In case of the local locatbn data, the dance of 
each human model is classified into one of the following 
three patterns by a motion data change section 44. 

JO 

(1 ) Same dance for all human models 

(2) Solo dance for each humari model 

(3) Solo dance for one human model and interlude 
dance for other human models 

IS 

[0066] Fig. 1 4 shows an interface display for a relation 
between the location data and the motion data in case 
of the bcal location data. In Fig. 1 4, three kinds of same 
dance for all human models, three kinds of solo dance 
20 for each human model (A, B, C), and two kinds of inter- 
lude dance for two human models are displayed. The 
user can indicate his desired kind of dance by inputting 
the motion file name (***. mtn). The relation data be- 
tween three dance patterns and the motion file is stored 
25 in memory space. 

[0067] In case of the general location data, a plurality 
of location patterns of the human model are prepared. 
The motion data are related with each location pattern. 
Fig. 15 shows an interface display for a relation between 
30 the locatbn data and the motion data in case of the gen- 
eral location data. As for the three location patterns pre- 
pared, the user can indicate the kind of dance of each 
human model A, B, C by inputting the motion file name 
(***. mtn). As for a location pattem other than the pre- 
ss pared three location patterns, the user can indicate the 
kind of dance of each human model A, B, C by inputting 
the motion file name (***. mtn). For example, as for one 
location pattern, assume that same motion file names 
are input for each human model A, B, C. If a location 
^0 pattern of three human models A, B, C is matched with 
the one bcation pattern, each human model dances in 
the same manner. The relation data between the loca- 
tion pattern and the motion file are stored in memory 
space. 

45 [0068] In Fig. 13, the location change section 43 
changes the bcation of the human model in virtual 
space. In this case, the content of the two dimensional 
array [x][y] stored in the location data memory section 
3 changes. In addition to this, the location data de- 

so scribed by the two-dimensional array is mapped to a po- 
sition coordinate in three-dimensional space to display 
the human model. 

[0069] In the first embodiment, the user interactively 
changes the location of the human model by key input. 
55 The user can move one human model on the two-dimen- 
sbnal lattice by an arrow key of a keyboard. Figs. 16A 
and 1 6B show an example of moving of the human mod- 
el A on the two-dimensional lattice by arrow key of the 
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keyboard. In the two-dimensional anray [x][y], " key 
corresponds to "+1 ■ along x direction, " key corre- 
sponds to "-1 ■ along x direction, ' T" key corresponds to 
"+1 " along y direction, " I' key corresponds to "-1" along 
y direction. As shown in Fig. 16A, assume that the hu- 5 
man model A is located at array [2][2] (=1) on two-di- 
mensional lattice. In case of pushing " key, the kx^a- 
tion data of the human model A is changed as shown in 
Fig. 16B (array [2]t2]=0, array [2][1]=1). When the hu- 
man model A is moved by above-mentioned method, 
the human models B, C are changed. As a result, all 
location data changes. 

[0070] Fig. 1 7 is a flow chart of moving processing of 
other human model by probability when one human 
model is moved by indication of the user. In Fig. 17, a 
figure "id=1 " represents the human model A; a figure "id- 
2" represents the human model B; and a figure ''ld=3'' 
represents the human model C. First, the human model 
A (id=1) is moved, the value of the new location array 
of the human model is "1 * and the value of the previous 
location array of the human model Is "0" (S1 71 ). In Fig. 

17, x-brg [id] and y-org [id] represent original position of 
each human model on two-dimensional lattice. Next, as 
for the human model B (ld=2)(S1 72), a number of lattice 
points movable for the human model B is cateulated as 
•neighboring points of original position of the human 
model B (S173). In this case, the neighboring point rep- 
resents eight lattice points surrounding the original po- 
sition. In case of moving the human mo6e\ by probability, 
the lattice point on which other human model is already 
located is a non-movable point. In Fig. 168, the human 
model B located at array [1][2] can move one of array 
[1][1], array [2][2], array [11[3], and array [2][3]. Next, 
probability of the lattice point movable for the human 
model B (id=2) is calculated (SI 74). In this case, the 
probability is 1/4 because the number of movable lattice 
points is four. Then, the human model B is moved ac- 
cording to the probability and predetermined random 
number (SI 75). As for the human model C (id-3) (S1 76), 
the moving control is executed in the same way (S172). 
[0071] Next, a method for mapping the location data 
described by two-dimensional array [x][y] to a position 
coordinate in three-dimensional space to display the hu- 
man model is described, in the three-dimensional 
space, XZ plane Is regarded as the ground shown in Fig. 

1 8. In this case, a direction of height of the human model 
is parallel to y axis. The mapping from two-dimensional 
array (x, y) to actual display space (X, Y, Z) is calculated 
by following equation using movement and scale con- 
version. 

X=(x-2)*5 



Y=0 



2=(y.2)*5 

[0072] The location change of the human model is not 
limited to key input of the user. For example, timing of 
the location change is periodically obtained using time 
data in the MIDI signal. Otherwise, as for some channel 
data, the location change is executed by change of mu- 
sical inten/al or intensity of the channel data as a key 
In short, timing of the location change Is automatically 
obtained from sound data. 

[0073] In Fig. 13, when the location change section 
43 changes the location of the human model in virtual 
space, a motion data change section 44 evaluates the 
changed location data and changes the motion data as- 
signed to the human model according to the motion data 
related by the location data / motion data relation section 
42. As an evaluation method of \ocsl\or\ of the human 
model, both the local tocatbn data and the general lo- 
cation data are explained as examples. 
[0074] Fig. 1 9 is a flow chart of evaluation processing 
by the local location data. In this method, it is decided 
that other human models are located neighboring a lat- 
tice point on whrch one human model is located on the 
two-dimensk)nal lattice. First, it is decided that other fig- 
ures (character model) are located at neighboring lattice 
points of each character model (SI 91). If the other fig- 
ures are located at neighboring lattice points, the value 
of parameter e is incremented by 1 (Si 92). If the other 
figures are not located, a parameter "id-solo" is as- 
signed to the character model as an isolated model 
(SI 93). As a result, motion of each character model is 
detemiined by the value of parameter e (SI 94). 
[0075] As shown In Fig. 20, in a situation that three 
human models A, B, C are adjacent each other, the val- 
ue of parameter e is "3". Therefore, same motion 
(dance) is assigned to three human models in order to 
keep a predetermined space form. In this case, cooper- 
ation in synchronization with same dance is represent-, 
ed. The kind of same dance is detemiined as one of 
three solo dances indicated by the location data / motbn 
data relation section 42 in Fig. 14. 
[0076] Fig. 21 shows a situation that one human mod- 
el A is isolated and other two human models are adja- 
cent each other. In this case, solo motion (dance) is as- 
signed to the one human model A. The value of param- 
eter e is "2". Therefore, interlude dance during the solo 
dance is assigned to the two human models B, C in order 
to represent cooperative dance by space gap. The kind 
of the solo dance is determined as the motion file as- 
signed to the human model A indicated by the location 
data / motion data relatton section 42 in Fig. 1 4. The kind 
of the interlude dance is determined as one of two mo- 
tion files Indicated by the location data / motion data re- 
lation section 42 in Fig. 1 4. 

[0077] As shown in Fig. 22. in a situation that three 
human models A, B, C are respectively isolated, the val- 
ue of parameter e is '0'. Therefore, solo dance is as- 
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signed to each model A, B, C in order to accent the co- 
operation of dance. The kind of solo dance ts deter- 
mined as the motion file assigned to each human model 
A, B, C indicated by the location data / motion data re- 
lation section 42 in Fig. 14. 

[0078] On the other hand, as an evaluation method of 
general location data, a matching between the location 
pattern prepared In Fig. 15 and the two-dimensional ar- 
ray [x][y] as changed location is executed. First, the lo- 
cation of the human model Is searched in xy space of 
the two-dimensional array [x][y]. If the searched location 
of the human model matches one of the location pat- 
terns in Fig. 15, the motion file related with the one lo- 
cation pattern is assigned to each human model A. B, C. 
[0079] In Fig. 13, a motion generation section 45 ob- 
tains tempo data from the sound parameter extracted 
by sound data extraction section 1 and moves the figure 
(character model) according to the tempo data with the 
motion data assigned to the figure. Fig. 23 is a flow chart 
of processing of the motion generation section 45. As 
mentioned-above, in the first embodiment, motion of the 
human mo6e\ is generated by a method of key frame 
animation. First, the tempo data is obtained from the 
sound parameter (S231). In order to synchronize the 
motion of the human with the tempo data, the number 
of divisions of interpolation between two poses of the 
motion is calculated using the tempo data (S232). By 
using the number of divisions of interpolation, Interpo- 
lated poses between two poses of the motion are cal- 
culated in order (S233). By drawing the pose and the 
interpolated motion in order, the motion of the human 
model is represented as animation (S234). This 
processing is repeated at an interval by generation of 
timer event, such as a timer of "40 ms' inten/al. Herein- 
after, a method to calculate the number of diviston of 
interpolation is explained. For example, the tempo data 
obtained from MIDI signal Is "120" representing 120 
times of tempo per one minute. Accordingly, time Inter- 
val of one tempo is "6000 (ms) / 1 20 = 500(ms)'. In the 
first embodiment, the time inten^al between poses de- 
scribed in the motion file is prescribed by a time inten^al 
of one tempo in order to synchronize the motion with the 
tempo. Fig. 24 shows an example of a time chart repre- 
senting a series of poses described in the motion file in 
case of tempo "120". As shown in Fig. 24, one period of 
the motion starts at pose PI , ends at pose P8, and re- 
turns to pose P1. The one period is totally represented 
as eight pose sections. In this case, a time interval (500 
ms) of one tempo is determined as an inten/al of two 
pose sections P1-P3. P3-P5, P5-P7, P7-P1. A number 
N of division of interpolation of one pose section (for ex- 
ample, P1-P2) is calculated by using "tempo" and the 
interval (40 ms) of timer as follows. 

N=((60000 / tempo) / 2) / 40=6 

[0080] In this way. the human model is moved in syn- 



chronization with the tempo. Therefore, each human 
model dances uniformly, and cooperation synchronized 
with time is represented. 

[0081] Next, a first modification of the editing section 
s 4 is explained. In the first modification, in addition to the 
above-mentioned editing function of the editing section 
4. channel (musical instrument) data of MIDI signal input 
from the sound data extraction section 1 is related with 
the character model, and the relation data is changed. 
10 At a timing to play the related channel, new motion re- 
lated with the character model Is derived at an interval 
of two poses synchronized with the tempo. In short, var- 
• ious kinds of motion of the character wo6e\ are edited / 
generated. 

IS [0082] Fig. 25 is a block diagram of the editing section 
4 according to the first modification. In Fig. 25, the figure 
data selection section 41 , the location data / motion data 
relation section 42, the location change section 43, the 
motion data change section 44 are same as each sec- 

20 tion in Fig. 1 3. A sound data / motion data relation sec- 
tion 46 relates the channel data and the motion data with 
the character model. Fig. 26 is an interface display of 
the sound data / motion data relation section 46. As for 
the human body character (figure), MIDI channel 

2S (number) and motion file name (***. mtn) are Input by 
the user, and related with the human body character 
This relation data is stored in memory space. MIDI chan- 
nel number "1 •-16" respectively corresponds to a musi- 
cal instrument used for a tune. Each channel is sounded 

30 at a peculiar (different) time during playback. In this in- 
terface, if a different channel is assigned to each human 
model, each human model respectively moves as an ad- 
lib at a different timing in an inten/al of the pose synchro- 
nized with the tempo. In short, each human model exe- 

3S cutes its special motion at its peculiar timing. If the same 
channel is assigned to each human model, each human 
model moves as the ad-lib at the same timing in an In- 
terval of the pose synchronized with the tempo. In the 
first nrKxIification, cooperation of synchronization or time 

40 lag of the motion is represented in case of ad-lib dance 
or motion of the human model. 
[0083] Fig. 27 is a flow chart of processing of the mo- 
tion generation section in the first modification. First, in 
addition to the tempo data obtained from the sounddata, 

4S channel data (note-on) indicating a channel number to 
play is obtained (S271). If the channel related with the 
human model (A, B, C Is represented as "id") is turned 
on (for example, a channel number "4" related with the 
human model A is turned on ) (S272), new motion is de- 

50 rived from the pose data synchronized with the tempo 

(5273) . The new motion Is derived from a motion file 
related to the human model A by the sound data / motion 
data relation section, If the channel related with the hu- 
man model is turned off, a number of division of inter- 
ns polatlon is calculated in the same way as in Fig. 23 

(5274) . and interpolated poses are generated (S275). 
[0084] Fig. 28 shows an example of a time chart rep- 
resenting new motion derived In a pose inten^al synchro- 
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nized with tempo. In Fig. 28, the derived motion consists 
of four poses (Pdl , Pd2, PdS, Pd4). At a timing of the 
channel's turning on, a section between the last pose 
Pe synchronized with tempo and a start pose Pdl of de- 
rived motion is interpolated by a number of division •2' 5 
(interpolation A). Next, each pose section (Pd1-Pd2, 
Pd2-Pd3, Pd3-Pd4) is interpolated by the number of di- 
visbn "2". A section between the last pose Pd4 of de- 
rived motion and a start pose Ps synchronized with tem- 
po is interpolated by the number of division "2' (interpo- 
latlon B). In this case, the start pose Ps is a pose inter- 
polated for ten times (the number of division '2" x "5" 
times) from the last pose Pe. As shown in the interpola- 
tions A and B of Fig. 28, at the start pose Pdl and the 
end pose Pd4 of derived motion, animation interpolated '5 
for the pose synchronized with tempo is generated. 
Therefore, the motion of the human model is smoothly 
displayed in a connected, section between original mo- 
tion and the derived motion. Furthermore, the start pose 
Ps Is determined by interpolating the total number of di- 20 
visions of the derived motion from the last pose Pe. 
Therefore, at an interyal between the last pose Pe and 
the start pose Ps, the original motion synchronized with 
tempo Is continuously kept without deviatbn between 
the tempo and the pose. In short, even if the derived 25 
motion of one human model is generated, this derived 
motion of the one human model and the motion of other 
human nnodels are always synchronized with the tempo. 
[0085] Next, a second modification of the editing sec- 
tion 4 Is explained. In the second modification, in addi- so 
tion to the editing function of the first modification, when 
the user (creator) interactively changes the motion data 
of one figure (one character model), the motion data of 
other figures (other character models) changes in pro- 
portion to the change of the motion data of the one fig- 5S 
ure. in short, the motion data of a plurality of figures are 
totally edited / generated. 

[0086] Fig. 29 is a block diagram of the editing section 
4 according to the second modification. |n Fig. 29, the 
figure data selection section 41 , the location data / mo- 40 
tlon data relation section 42, the location change section 
43. the sound data / motion data relation section 46 are 
same as each section in Fig. 25. An operation / motion 
data relation section 47 relates the interactive operation 
of the user with the motion data of the character model. ^5 
In the second modification, key input from the keyboard 
is used as interactive operation means of the user. 
[0087] Fig. 30 is an interface display to relate keys of 
the keyboard with the motion data. In Fig. 30, using four 
keys H, J, K, L. the motion file (***. mtn) is input to relate 50 
the motion data with the key. A motion operation sectk>n 
48 interactively changes the motion data of the human 
model by the user. In Fig. 30, the user can operate four 
keys H. J, K, L related by the operation / motion data 
relation sectbn 47. In this case, the operation / motion ^ 
data relation section 47 may relate an image of move- 
ment or gesture of actual person with the motion data 
of the human model. In case of image recognitbn (for 



example,* template-matching) through CCD camera, in- 
teractive operation by the movement or gesture of the 
person is possible. 

[0088] The motion data change section 44 assigns 
the motion data operation by the motion operation sec- 
tion 48 with one human model. After determining the rel- 
ative bcation between the one hunnan model and other 
human models, the motion data of the other human 
models are changed. 

[0089] Fig. 31 is a flow chart of processing of the mo- 
tion data change section 44 in case of input operation 
from the motion operation section 48. First, motion file 
X related with input key Is assigned to the human model 
A (id=1 ) (S31 1 ). Next, it is decided that the human model 
A is located neighboring a human model B (id=2) on the 
two-dimensional lattice (S312, S313). If the human 
model A Is located along a lateral direction (X direction) 
of the human model B, the motion X is assigned to the 
human model B as 'detay-0', whbh Is executed at the 
same timing as the motion of the human model A 
(S314). In other cases, the motion X is assigned to the 
human model B as 'delay-1 which is executed at delay 
timing from the motion of the human model A (S315). 
Hereinafter, as for a human model C (kJ-3), the motion 
is assigned after ctetemilning the relative kx;ation be- 
tween the human model A and the human model C in 
same way. 

[0090] As mentloned-above, in the second modifica- 
tion, the human nwdels B. C execute the motion X at 
the same timing or delay timing for the motion X of the 
human model A according to the location status. There- 
fore, cooperation by synchronization or delay in organ- 
ized dance is represented. 

[0091] The motion generation section 45 generates 
motbn of each human model at the timing (same or de- 
lay) assigned by the motion data change section 44 in 
the same way as in the first modification. In Fig. 28, a 
timing of channel on is replaced by a timing of key input 
of the user. As for the motion of the human model (de- 
lay=0). the motion generation section 45 generates the 
motion at the timing of key input. As for the motion of 
the hunf)an model (delay=1 ), the motion generation sec- 
tion 45 generates the motion at a delay timing of several 
frames from the key input. 

[0092] In Fig. 1 , the drawing section 9 executes draw- 
ing according to the figure data and the motion data re- 
lated with the location data and sound data by editing 
section 4. The sound output section 2 extracts the sec- 
ond data such as tune play data from MIDI signal and 
plays the sound data. The display section 10 converts 
the content drawn or updated by the drawing sectbn 9 
to video signal and displays the drawn content as ani- 
mation. Fig. 32 shows one scene of displayed anima- 
tion. 

[0093] As mentioned-above, in the first embodiment, 
a relation among the sound, the figure, and the motion 
is interactively edited while confirming the sound andthe 
vkjeo. Therefore, various videos are created for one 
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piece of music. 

[0094] Furthermore, the motion of each figure (char- 
acter model) is controlled according to space location of 
a plurality of figures. Therefore, a plurality of human 
models change their forming in space and time to the 
music, and each human model acts by changing the mo- 
tion according to the forming, in short, space-coopera- 
tion in organized dance is represented as animation. 
[0095] Furthermore, timing of the motion related with 
a plurality of figures is synchronized or shifted. There- 
fore, a plurality of human models move at the same tim- 
ing or propagate the motion at a delay timing in order. 
In short, time-cooperation in organized dance is repre- 
sented as animation. 

[0096] Next, a second embodiment of the present in- 
vention is explained. Fig. 33 is a block diagram of the 
animation creation apparatus according to the second 
embodiment. In Fig. 33, the animation creation appara- 
tus includes the sound data extraction section 1, the 
sound data output section 2, the location data memory 
section 2. the editing section 4, the figure data memory 
section 5, the motion data memory section 6, the pose 
data memory section 7, the motion data creation section 
8, the drawing section 9, the display section 10, and a 
face image memory section 50. Each section except for 
the editing section 4 and the face image memory section 
50 is same as the first embodiment in Fig. 1 . 
[0097] The face image merno^ section 50 stores a 
face image and a back image of a head to synthesize 
on the head part of human model. From a person's face 
image input by a digital camera, scanned by a scanner, 
or drawn by a paint tool, the face image is obtained by 
eliminating a background image. As the face image 
without the background infiage, for example, a user 
paints over the background area except for the face area 
(including hair part) as white color (R:255. G:255, B:255) 
by a paint-tool. In this way, the face image without the 
background area is obtained. From a person's back im- 
age of the head input by the digital camera, scanned by 
the scanner, or drawn by the paint tool, the back Image 
of the head is obtained by eliminating the background 
image. The back image of the head may be obtained by 
drawing a face area of the face image in the color of hair. 
[0098] As shown in Fig. 34, the face Image memory 
section 50 stores a series of face images (000. bmp, 
001 . bmp, •••) with the back image of the head (Back 
000. bmp) as one set. Each of the face images and the 
back image of the head is a static Image file such as 
bitmap format (BMP). As the series of face images, ten 
frames each of which facial expression is different are 
used. 

[0099] Fig. 35 is an example of a file to store the face 
image and the back image of the head as one set. In 
short, the series of the face images and the back image 
of the head are described in this file. A description 'FMV 
[nframe 10]' represents that the series of the face im- 
ages consists of ten frames. A description 'Frame[id 0 
fname ' 000. bmp"]" indicates 1st face image file. Fol- 



bwing description of 'Frame" indicates the series of 
face image file. The fast description 'BackFrame [id 0 
fname ' BackOOO. bmp']' indicates the back image file 
of the head. In the second emtxwjiment, above-men- 
5 tloned file Is called Ff\/IV file by using an extension 
fmv". 

[01 00] The editing section 4 edits / generates various 
motbn of display content and CG character. For exam- 
ple, the editing section 4 selects a desired FMV file from 
10 the face image memory section 50, puts or renders the 
face image on a head part of CG character, changes CG 
character, and changes a relation between the sound 
data and the motion data. 

[0101] Fig. 36 is a block diagram of the editing section 

IS 4 of the second embodiment. In Fig. 36, each section 
except for the figure data selection 41 , a figure change 
section 51 , and a face image put section 52 is the same 
as in the first embodiment in Fig. 29. The figure data 
selection section 41 selects a kind of CG character and 

20 FMV file of the face image to put on head part of the CG 
character. As for selection of CG character and FMV file, 
a dialog box for selecting character shown In Fig. 37 is 
used. This diabg box is activated by pointing a mouse- 
cursor on CG character and clicking. In order to select 

25 CG character, the user describes a file name of desired 
CG character in a column of character file in Fig. 37. In 
order to select FMV file, the user describes a FMV file 
name in the FMV file column in Fig. 37. 
[01O2] In Fig. 36. the figure change section 51 replac- 

30 es the indicated CG character by FMV file or CG char- 
acter selected by the figure data selection section 41. 
Figs. 38A, B, C show one example of CG character to 
put on the face image. Fig. 38A shows CG character 
indicated by the user. First, as shown in Fig. 38B, a head 

3S part of CG character is replaced by a plane plate. Next, 
as shown in Fig. 38C, a first image of FMV file is put on 
the plane plate by texture-mapping. In this case, a back- 
ground pari drawn by white color of face image is trans- 
parentized. As for this transparence, for example, tex- 

40 ture functbn of Direct 3D as API of three-dimensional 
CG is used while transparency of white part is effective. 
In this way, the face image including the face area and 
a transparent background is mapped to the head part of 
CG character 

45 [0103] In Fig. 36, as for the plane plate of the head 
part of CG character replaced by the figure change sec- 
tion 51, the face Image put section 52 periodically puts 
the face image and the back image of the head of FMV 
file by changing each image. 

so [01 04] Fig. 39 is a flow chart of processing of the face 
image put section 52. First, pose data of CG character 
generated by the motion generation section 45 and 
camera data representing position / direction to view the 
three-dimensional scene (front position /direction of the 

55 display) are obtained (S391). A method for putting the 
face image and the back image of the head is deter- 
mined by the pose data and the camera data. The pose 
data represents position / direction of each part (head, 
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chest, etc) of CG character. As a basic processing, ten 
frames of the face images is periodically changed and 
put on the plane plate by a timer of "100 ms" interval. 
After the tenth face image is put (S392), the first face 
image is put next as repeat processing (S393). s 
[010S] Next, it is determined whether the back of the 
human model of the CG character faces the camera po- 
sition (user's view position). An inner product between 
a vector Va along a direction of chest of the human mod- 
el and a vector Vb linked from a chest position to the 
front position of the display is calculated. If "Va-Vb > =0", 
the human model is decided to face to the front position. 
If 'Va-Vb < 0", the back of the human nrxxiel is decided 
to face to the front position. If the human model faces to 
the front position of the display, C-th face image is put 
on the plane plate of the head part of the human model 

(5395) . If the back of the human model faces to the front 
position of the display, the back image of the head is put 
on the plane plate of the head part of the human model 

(5396) . A direction of the plane plate of the head part is 
changed according to a direction of the human model 
for the front position of the display (S397). In order to 
always turn the face image to the front position of the 
display, a direction of the plane plate of the head part is 
corrected as a vector linked from the head position of 
the human model to the front position of the display 
[01 06] Fig. 40 shows one example of the human mod- 
el on whteh the back image of the head is put. By above- 
mentioned processing, the face image of different ex- 
pression Is periodically changed in synchronization with 
pose of motion of the character model. Furthermore, the 
face Image is changed to the back image of the head 
according to a direction of the character model. 
[01 07] In the second embodiment, the figure data se- 
lectbn section 41 relates one FMVfile with one CG char- 
acter. However, as modification, a plurality of FMV files 
may be related and changed in order according to key 
operation of the user or the sound data. In this case, for 
example, an FMV file including a laugh expression and 
an FMV file including a surprise expression are pre- 
pared and selectively changed at a timing of sound gen- 
eration. In this way, the face expression of the human 
model is dynamically changed from the laugh expres- 
sion to the surprise expression. 
[0108] Furthermore, In the second embodiment, the 
face Image is put on the head part of the human model. 
However, a dress part image is extracted from the upper 
half of the body of the person's image. In this case, the 
background area is painted by white color. Then, the ex- 
tracted image Is put on a chest part of the human model 
as shown in Fig. 41 . In the same way as the back Image 
of the head, the dress part of the extracted image is 
painted by color of the dress and used as the back im- 
age. When the human model turns tail to the camera 
position, the back image is put on the chest part. 
[0109] Next, a third embodiment of the present inven- 
tion is explained. Fig. 42 is a block diagram of the ani- 
mation creation apparatus according to the third embod- 
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iment. As shown In Fig. 42, the animatiori creation ap- 
paratus Includes the sound data extraction section 1 , the 
sound data output sectton 2, the location data memory 
section 3. the editing section 4. the figure data memory 
section 5, the motion data memory section 6, the pose 
data memory section 7, the drawing section 9, the dis- 
play sectk>n 10. the face image memory section 50, a 
face image extraction sectton 60, a back image gener- 
ation section 61 . 

[0110] In the third embodiment, theface image extrac- 
tion section 60 and the back image generation section 
61 automatically generates the face image and the back 
image of the head. These images are selectively put on 
the head plate of the human model nrraving in synchro- 
nization with the music. In Fig. 42, each section except 
for the face image extraction section 60 and the back 
Image generation sectton 61 is same as In the second 
embodiment in Fig. 33. 

[0111 ] The face image extractbn section 60 automat- 
ically extracts a face part from a person's image. As a 
method for automatically extracting the face part from 
dynamic image, for example, a fractal extraction method 
is used In the fractal extraction method, the face image 
(including hair part) from which the background area Is 
eliminated Is extracted at a processing speed of ten 
frames per second. In short, a series of face images as 
ten frames is extracted for one second. Each face image 
is stored as one static image file (BMP) while the elimi- 
nated background part is changed as white color area 
(R;255, G:255, 8:255). The back image generation sec- 
tion 61 automatically generates the back image of the 
head of a person based on one face Image extracted by 
the face image extraction section 60. For example, In 
the first frame In ten frames, the face area is painted the 
cobr of hair. In this case, the darkest color in the face 
image is used as the color of hair. Furthermore, it often 
happens that a boundary of the back of the head is ir- 
regular because of protrusions such as an ear There- 
fore, a processing of smoothing is executed for the 
boundary. The nape or low part (several bits part) of the 
back of the head area is painted white in order to cut 
along a horizontal direction. The back image of the head 
is preserved as a static image file (BMP) in the same 
way as the face image. These face images and the back 
image of the head as ten frames are stored as one file 
in the face image memory section 50. As mentioned- 
above, the file name is described In FMVfile. 
[0112] As mentioned-above, the third embodiment Is 
applied to the human model's dancing in virtual space. 
However, the third embodiment is also applied to a chat 
or a virtual mall of which a plurality of users owns jointly 
from a remote place. For example, the human model 
(three-dimensional character) as a representative of 
each user converses or walks in virtual space. In this 
case, the face Image or the back image of the head is 
synthesized with the head part of the human model as 
if the user himself converses or walks in the virtual 
space. The face image may be nrK>ved In synchroniza- 
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tion with text conversation or walking in order to repre- 
sent different motbn for each user. 
[0113] Furthermore, in case of music-distribution, a 
character model including singer's face image with mu- 
sic data is supplied to a virtual system. If the user adds 
hisface image to this virtual system, he can easily create 
a situation that the human model himself dances with 
the singer. Alternatively, face images of audience in a 
hall or in a remote place are extracted from input images 
during a live concert. On a background display of a stage 
in the hall, the human models, including the face imag- 
es, virtually dance in synchronization with the singer's 
music. In this way, a new performance is realized. 
[0114] As mentioned-above, in the third embodiment, 
a plurality of object images and the back image of the 
object image are selectively synthesized with the char- 
acter model at a predetermined inten/al or by a direction 
of the character model. In this case, a concert-hall pres- 
ence that the user is dancing with the character model 
is easily generated. Concretely speaking, a plurality of 
face images and the back image of the head are selec- 
tively put on the head part of the human model. There- 
fore, animation whose face expression changes in co- 
operation with the motion is easily created. Especially, 
a rotation motion of the human model such as a balleri- 
na, that the face always turns to the front and the back 
of the head turns to the front at one moment only, is eas- 
ily created. Therefore, the user can feel a concert-hall 
presence. 

[01 1 5] Furthermore, in the third embodiment, the face 
image and the back image of the head are automatically 
generated from a dynamic image such as video. There- 
fore, the user's load to create the face image and the 
back image of the head is omitted and animation-motion 
including change of face expression Is easily realized. 
[0116] Furthermore, the upper half of the body is ex- 
tracted and put on the human model in addition to the 
face image. In this case, the virtual space including more 
concert-hall presence is easily realized. 
[0117] A memory can be used to store instructions for 
performing the process described above. The process 
may be performed with the aid of a general purpose 
computer or microprocessor. Such a memory can thus 
be a CD-ROI^, floppy disk, hard disk, nr^gnetic tape, 
semiconductor memory, and so on. 
[0118] Furthermore, according to indication of pro- 
gram installed from the memory to the computer, OS 
(operation system) on the computer, database manage- 
ment software and f^W (middle ware) such as network 
may execute a part of each processing to realize the 
present invention. 

[01 1 9] Furthermore, the memory of the present inven- 
tion is not limited to a medium Independent from the 
computer. The memory includes a medium to store a 
downloaded program transformed by LAN or Internet. 
[0120] Furthermore, the memory medium is not limit- 
ed to one unit. In case the processing of the present 
inventk>n is executed by programs stored in a plurality 
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of memory mediums, the plurality of memory mediums 
are included in the present invention. 
[0121] In the present invention, the computer exe- 
cutes each processing of the present embodiment ac- 

5 cording to the program stored in the memory medium. 
This computer may be one apparatus such as a person- 
al computer, or a system of which a plurality of the ap- 
paratuses are connected by a network. 
[0122] Furthermore, in the present invention, the 

10 computer is not limited to a personal computer. The 
computer may be a microcomputer or a processor in- 
cluded in an information processing apparatus. In short, 
the processor and the apparatus executable the func- 
tion of the present invention by a program are generi- 

'5 cally called the computer. 

[01 23] Other embodiments of the invention will be ap- 
parent to those skilled in the art from consideration of 
the specification and practice of the invention disctosed 
herein. It is intended that the specification and examples 

20 be considered as exemplary only, with the* true scope 
and spirit of the invention being indicated by the follow- 
. ing claims. 

25 Claims 

1 . Animation creation apparatus, comprising: 

sound data extraction means for extracting a 
30 predetermined parameter from sound data; 

figure memory means for storing a plurality of 
character models to be displayed; 
motion data memory means for storing motion 
data of each of the plurality of character mod- 
35 els; 

location data memory means for storing loca- 
tion data of each of the plurality of character 
models in virtual space to be displayed; 
indication means for indicating a change in the 
40 location data of at least one of the plurality of 

character models; 

editing means for changing the motion data of 
each of the plurality of character models to be 
in cooperatbn with each other according to the 
45 predetermined parameter and the changed k?- 

cation data; and 

display means for displaying the plurality of 
character models each moving according to the 
changed motion data. 

so 

2. The animation creation apparatus according to 
claim 1, 

wherein the motion data includes a plurality of 
pose data at predetermined intervals showing char- 
55 acter model movement, a series of the pose data 
periodically representing one motion of the charac- 
ter model. 
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3. The animation creation apparatus according to 
claim 2. 

wherein the plurality of pose data Is inputted 
through said indication means by a user while the 
character model is displayed. 

4. The animation creation apparatus according to 
claim 1, 

wherein said editing means relates the loca- 
tion data of each character model with the motion 
data of each character model, the relation between 
the location data and the nrK)tion data being local 
location data or general locatbn data. 

5. The animation creation apparatus according to 
claim 4, 



location pattern matched with the location data of 
the plurality of character OKxiels. 

9. The animation creatiori apparatus according to 

5 claim 2, 

wherein said editing means calculates a 
number of divisbns of interpotatbn between every 
two pose data of the character model according to 
tempo data of the predetemnined parameter, the 

10 two pose data being synchronized with the tempo 
data, and creates pose data of each timing corre- 
sponding to the number of divisions between every 
two pose data as an interpolation drawing. 

15 10. The animation creation apparatus according to 
claim 9, 



wherein the local location data includes the 
same motion for all character models as a first 
selection item, different solo motion for each 20 
character model as a second selection item, 
and solo motion for one character model and 
interlude motion for other character models as 
a third selection item; and 
wherein the general location data includes a 2S 
correspondence relation between a location 
pattern for a plurality of character models and 
the motion data for each character model, a plu- 
rality of correspondence retatbns being prede- 
termined. 30 

6. The animation creation apparatus according to 
claim 1 , 



wherein said indication means indicates a cor- 
respondence between one character nrvodel 
and a channel of the sound data, and 
wherein said editing means generates derived 
motion data for the one character model be- 
tween an on-timing and an off-timing of the 
channel, the derived motion data including 
pose data. 

11. The animation creation apparatus according to 
claim 10, 

wherein said editing means interpolates a 
start period between the pose data of the on-timing 
and start pose data of the derived motion, and an 
end period between end pose data of the derived 
motion and the pose data of the off-timing. 



wherein when said indication means indicates 3S 
a change of the location data of at least one 
character model, 

said editing means selectively changes the lo- 
cation of other character models in proportion 
to the change of the location of the at least one 40 
character model. 

7. The animation creation apparatus according to 
claim 5, 

wherein said editing means selects the first 
selection Item if the location of all character models 
are close to each other, selects the second selec- 
tion item if the locations of each character model 
are dispersed, and selects the third selection item 
if the location of one character model is separated so 
from the location of neighboring character models. 

8. The animation creation apparatus according to 
claim 5, 

wherein said editing means compares the lo- ss 
cation data of a plurality of character models with 
each location pattern of the general location data, 
and selects the motion data corresponding to one 



12. The animation creation apparatus according to 
claim 2, 

wherein said indication means indicates a cor- 
respondence between one motion data and 
one character model, and 
wherein said editing means decides whether 
the location data of other character model is 
- close to the location data of the one character 
model. 

13. The animation creation apparatus according to 
claim 12, 

wherein said editing means assigns the one 
motion data to the other character model in syn- 
chronization with a timing of the one motion data of 
the one character model if the location data of the 
other character model is close to the location data 
of the one character model. 

14. The animation creation apparatus according to 
claim 13, 

wherein said editing means assigns the one 
motion data to the other character model by shifting 
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the timing of the one motion data of the one char- 
acter model if the location data of the one character 
model is not close to the location data of the one 
character nrKxIel. 

5 

15. A method for creating animation, comprising the 
steps of: 

extracting a predetermined parameter from 
sound data; 

storing a plurality of character models to be dis- 
played; 

storing motion data of each of the plurality of 
character models; 

storing location data of each of the plurality of 
character models in virtual space to be dis- 
played; 

indicating a change in the location data of at 
least one of the plurality of character models; 
changing the motion data of each of the plurality 20 
of character models to be in cooperation with 
each other according to the predetermined pa- 
rameter and the changed location data; and 
displaying the plurality of character models 
each moving according to the changed motion 2S 
data. 

16. A computer readable memory containing computer 
readable instmctions, comprising: 

30 

instruction means for causing a computer to ex- 
tract a predetermined parameter from sound 
data; 

instructbn means for causing a computer to 
store a plurality of character models to be dis- 35 
played; 

instruction means for causing a computer to 
store motion data of each of the plurality of 
character models; 

instruction means for causing a computer to 

store location data of each of the plurality of 
character models in virtual space to be dis- 
played. 

instruction means for causing a computer to in- 
dicate a change in the location data of at least 
one of the plurality of character models; 
instruction means for causing a computer to 
change the motion data of each of the plurality 
of character models to be in cooperation with 
each other according to the predetermined pa- so 
rameter and the changed location data; and 
instruction means for causing a computer to 
display the plurality of character models each 
moving according to the changed motion data. 

55 

17. Animation creation apparatus, comprising: 

figure menrK)ry means for storing at least one 



character model to be displayed; 

motion data memory means for storing motion 

data of the one character model; 

indication means for indicating motion of the 

one character; 

editing means for moving the one character ac- 
cording to the motion data in response to the 
indication; 

object image memory means for storing a plu- 
rality of object images to compose on the one 
character; 

image composition means for selectively com- 
posing each of the plurality of object images on 
the one character in synchronization with mo- 
tion of the one character; and 
display means for displaying the motion of the 
one character on which the object image is 
composed. 

18. The animation creation apparatus according to 
claim 17, 

wherein the motion data includes a plurality of 
pose data at predetermined inten^als showing 
character model movement, a series of the 
pose data periodically representing one motion 
of the character model; and 
wherein the plurality of object images are face 
images of the one character corresponding to 
each timing of the plurality of pose data, one of 
the plurality of object images back the back im- 
age of the head of the one character. 

19. The animation creation apparatus according to 
claim IB, 

wherein said image composition means peri- 
odically changes each of the plurality of face images 
to compose the one character in synchronization 
wrth each timing of the plurality of pose data to move 
the one character. 

20. The animation creation apparatus according to 
claim 19, 

wherein said editing means generates camera 
data representing the front direction of said dis-. 
play means along a user's line of sight, and 
wherein said image composition means de- 
cides whether the one character turns its back 
on the front direction according to the pose data 
and the camera data. 

21. The animation creation apparatus according to 
claim 20, 

wherein said image composition means com- 
poses the face image on a head plate of the one 
character if the one character is decided not to turn 
its back on the front direction, and composes the 
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back image of the head on the head plate of the one 
character if the one character is decided to turn its 
back on the front direction. 

22. The animation creation apparatus according to 
claim 21 , 

wherein sakj Image composition means 
changes a direction of the head plate of the one 
character in order to continually face to the front di- 
rection in synchronization with each timing of the 
plurality of pose data. 

23. The animation creation apparatus according to 
claim 16, 

further comprising a face image extraction 
means for respectively extracting the face im- 
age from a series of original images each in- 
cluding a human, and 

a back Image generation means for generating 
the back image of the head using at least one 
of the plurality of face images extracted by said 
face image extraction means. 

24. The animation creatbn apparatus according to 
claim 23. 
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played; 

Instruction means for causing a computer to 
store nrK)tion data of the one character model; 
instruction means for causing a computer to in- 
dicate motion of the one character; 
instruction means for causing a computer to 
move the one character according to the motion 
data In response to the indication; 
instruction means for causing a computer to 
store a plurality of object images to compose 
on the one character; 

instruction means for causing a computer to se- 
lectively compose each of the plurality of object 
Images on the one character in synchronization 
with motion of the one character; and 
instruction means for causing a computer to 
display the motion of the one character on 
which the object image is composed. 



wherein said object image memory means 
stores the plurality of face images as one of a 
plurality of face movie files, and 
wherein said Indication means selects one of 
the plurality of face movie files, the plurality of 
face images of the one face movie file being 
composed on the head plate of the one char- 
acter. 



30 



35 



25. A method for creating animation, comprising the 
steps of: 

storing at least one character model to be dis- 40 
played; 

storing motion data of the one character model; 
Indicating motion of the one character; 
moving the one character according to the mo- 
tion data In response to the indbatbn; 
storing a plurality of object images to compose 
on the one character; 

selectively composing each of the plurality of 
object images on the one character in synchro- 
* nizatlon with motion of the one character; and so 
displaying the motion of the one character on 
which the object Image is composed. 

26. A computer readable memory containing computer 
readable instmctlons, comprising: ss 



Instructton means for causing a computer to 
store at least one character model to be dls- 
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#POSE RLE 

Body{ 
nparts 15 

} 

Part{ 
id 0 

dir 0.549239 0.021 340 0.835393 
up -0.032454 0.999464 -0.0041 94 
pos 0.000000 0.308000 0.000000 

) 

Part{ 
id 1 

dir 0.5601 1 4 -0.305377 0.770076 
up 0.459378 0.888058 0.018035 
pos '0.51 8209 -0.609804 0.381 773 

) 

Part{ 
id 2 

dir 0.575887 -0.23631 7 0.782629 
up 0.457741 0.886391 -0.069174 
pos -1 .207795 -1 .945265 0.331440 

} 
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